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1.  Introduction 


Several  statistical  and  experimental  design  and  analysis  textbooks  cover  basic  and  advanced 
experimental  design  techniques  (Keppel  and  Wickens,  2004;  Montgomery,  2008;  Myers  and 
Montgomery,  2002;  Shadish,  Cook,  and  Campbell,  2002;  Winer,  Brown,  and  Michels,  1991)  as 
well  as  attempts  at  automated  experimental  design  (Mills,  Meyer,  Rickels,  and  Hoyland,  1982). 
What  they  do  not  provide,  and  what  is  needed,  is  a  computer-based  reference  (using  modem 
hardware  and  software  technologies)  that  incorporates  an  integrated  subset  of  currently  accepted 
basic  and  advanced  experimental  design  approaches  that  human  factors  and  ergonomics 
researchers  can  use  in  the  complex  system  research  environments  they  are  investigating. 

Procedural  guidance  and  reference  for  experimental  design  and  statistical  analysis  was  often 
summarized  in  hard  copy  tables  in  the  covers  of  textbooks  (Siegel  and  Castellan,  1988).  With 
the  advent  of  hyperlink  technology  that  can  embed  such  information  into  machine  independent 
Postscript  Definition  File  (PDF)  documents,  modem  computer  technology  provides  the  technical 
basis  for  the  development  of  an  interactive  desktop  tool  to  access  a  large  reference  resource  to 
assist  researchers  in  their  quest  for  efficient  and  effective  designs. 


2.  Description  of  the  CADRE  Tool 


The  research  on  command  and  control  (C2)  systems  is  a  real  case  example  of  how  the 
availability  of  an  experimental  design  and  analysis  tool  can  significantly  assist  researchers  in 
experimental  design  development  and  analysis  of  the  results.  Appropriate  experimental  designs 
for  complex  analyses  can  be  overwhelming  to  complete,  demanding  to  structure,  and  incomplete 
in  their  results  if  an  ineffective  experiment  is  followed.  Considering  the  needs  of  this  type  of 
highly  complex  study  provided  the  basis  for  developing  procedures  and  processes  to  ease  the 
burden  of  experimental  design  development  and  ensure  that  an  effective  design  is  selected  that 
minimizes  the  data  collection  effort  while  maximizing  the  validity  of  the  data. 

In  2003  the  U.S.  Army  Research  Laboratory  (ARL),  in  collaboration  with  the  Virginia 
Polytechnic  Institute  and  State  University  (Virginia  Tech),  sponsored  the  development  of  just 
such  a  tool,  the  Computer-Aided  Design  Reference  for  Experiments  (CADRE).  This  three-year 
effort  is  now  coming  to  fruition  and  the  resulting  interactive  experimental  design  and  statistical 
analysis  reference  resource  provides  a  highly  effective  aid  for  researchers  to  quickly  establish 
efficient  experimental  designs  for  their  work. 
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2.1  CADRE  Structure 

The  CADRE  tool  is  as  an  interactive  desktop  computer  tool  designed  to  aid  human  factors  and 
ergonomics  researchers  in  experimental  design  and  analysis.  This  tool,  with  the  full 
documentation  papers  describing  it,  is  on  a  CD  accompanying  this  report.  As  the  CADRE  tool 
was  developed  with  Government  funding  it  is  in  the  public  domain  and  there  are  no  distribution 
limitations  associated  with  it. 

The  human  factors  experimental  design  and  analysis  reference  material  in  CADRE  is  a 
compilation  of  two  technical  reports  (Slater  and  Williges,  2006;  Williges,  2006).  The  Williges 
report  provides  the  primary  reference  material  on  experimental  design  in  a  PowerPoint 
presentation  with  accompanying  notes.  Over  850  pages  of  experimental  design  and  analysis 
reference  material  for  human  factors  researchers  is  provided  in  this  report.  The  content  of  the 
reference  material  is  presented  in  a  numbered  outline  format  on  PowerPoint  slides,  and 
descriptive  text  together  with  appropriate  reference  citations  are  provided  for  the  content  of  each 
slide.  At  the  end  of  each  topic  presentation,  references  are  provided  to  the  current  scientific 
literature  for  supplemental  readings  that  are  pertinent  to  the  topic  presented.  The  companion 
report  by  Slater  and  Williges  is  an  appendix  to  Williges’  report  and  provides  over  200  pages  in 
SAS  of  program  files  (SAS,  2004),  output  files,  and  explanations  for  40  examples  of  statistical 
analyses  covered  by  the  2006  Williges  report. 

The  CADRE  tool  has  several  interactive  features  that  can  be  used  to  rapidly  locate  reference 
material  information.  These  features  include  extensive  bookmarks  within  the  PDF  file;  links 
between  the  reference  material  and  the  appendix  for  each  example  computational  problem;  and 
hyperlinks  from  the  CADRE  tool  to  the  SAS  desktop  interactive  statistical  program  for  each 
example  problem. 

2.2  CADRE  Topics 

The  CADRE  reference  material  covers  25  topics,  divided  in  to  five  major  sections,  including  an 
introduction  to  experimental  design;  supplemental  data  collection  design  and  analysis;  basic 
analysis  of  variance  (ANOVA)  designs;  advanced  ANOVA  designs;  and  empirical  model 
building  as  shown  in  table  1 . 
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Table  1.  Major  topics  in  CADRE. 


Overview 

Section  1.  Introduction  to  Experimental  Design 

Topic  1.  Research  Design  Process 

Topic  2.  Experimental  Designs 

Topic  3.  Basic  Statistical  Concepts 

Section  2.  Supplemental  Data  Collection  and  Analysis 

Topic  4.  Supplemental  Data  Collection  Methods 

Topic  5.  Analysis  of  Nominal  Scale  Data 

Topic  6.  Analysis  of  Ordinal  Scale  Data 

Topic  7.  Summary  of  Supplemental  Data 

Section  3.  Basic  Analysis  of  Variance  (ANOVA)  Design 

Topic  8.  Introduction  to  ANOVA 

Topic  9.  ANOVA  Summary  Table  Components 

Topic  10.  Between-Subjects  ANOVA  Designs 

Topic  11.  Analysis  of  Comparisons  and  Interactions 

Topic  12.  Within- Subjects  ANOVA  Designs 

Topic  13.  Mixed-Factors  ANOVA  Designs 

Topic  14.  Summary  of  Basic  ANOVA 

Section  4.  Advanced  ANOVA  Designs 

Topic  15.  Introduction  to  Advanced  ANOVA 

Topic  16.  Hierarchical  ANOVA  Designs 

Topic  17.  Blocking  ANOVA  Designs 

Topic  18.  Fractional  Factorial  ANOVA  Designs 

Topic  19.  Analysis  of  Covariance  (ANCOVA) 

Topic  20.  Summary  of  Advanced  ANOVA 

Section  5.  Empirical  Model  Building 

Topic  21.  Introduction  to  Empirical  Models 

Topic  22.  Multiple  Regression 

Topic  23.  Central-Composite  Designs  (CCDs) 

Topic  24.  Sequential  Experimentation 

Topic  25.  Summary  of  Empirical  Models 

2.2.1  Topic  1.  Research  Design  Process 

Topic  1  presents  the  overall  research  design  process.  First,  the  various  stages  of  research  are 
shown  in  a  flow  diagram.  Next,  six  critical  aspects  of  this  process  are  highlighted,  from  the 
research  problem  to  research  reports.  As  with  all  subsequent  topics  covered  in  the  reference 
material,  this  topic  concludes  with  a  summary  followed  by  suggestions  for  supplemental 
readings  for  in-depth  coverage  of  the  material  covered  in  this  topic. 
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2.2.2  Topic  2.  Experimental  Designs 

Topic  2  is  an  introduction  to  experimental  design.  It  begins  with  an  overview  of  various 
conditions  that  can  threaten  the  validity  of  the  results  of  experiments  and  then  offers  a  general 
notation  for  designating  designs.  Two  general  experimental  design  alternatives  are  presented, 
and  the  case  is  developed  for  always  choosing  randomized  experimental  designs,  if  possible. 

2.2.3  Topic  3.  Basic  Statistical  Concepts 

Topic  3  summarizes  basic  statistical  concepts  that  are  fundamental  to  experimental  design.  This 
reference  material  is  not  a  tutorial  on  basic  statistics,  but  rather,  the  material  highlights  and 
summarizes  key  statistical  concepts.  It  is  assumed  that  users  of  this  reference  material  already 
have  a  background  in  introductory  descriptive  and  inferential  statistics.  Consequently,  this  topic 
is  designed  as  a  review  of  basic  concepts  without  providing  detailed  descriptions  or 
mathematical  derivations.  Users  should  refer  to  a  textbook  such  as  Walpole’s  latest  edition  on 
introductory  statistics  (Walpole,  Myers,  Myers,  and  Ye,  2002)  for  details  on  the  concepts 
summarized  in  this  topic  if  they  are  not  familiar  with  them.  The  concepts  of  sampling 
distributions,  the  F-distribution,  and  statistical  hypothesis  testing  are  critical  to  understanding  the 
experimental  design  topics  covered  in  this  reference. 

2.2.4  Topic  4.  Supplemental  Data  Collection  Methods 

Topic  4  deals  with  supplemental  data  collection  that  augments  data  from  experimental  designs. 
Quantitative  supplemental  data  are  very  important  for  understanding  and  interpreting  research 
results.  Most  supplemental  data  are  analyzed  by  nonparametric  analyses  as  opposed  to  the 
parametric  analyses  used  on  the  major  dependent  variables  manipulated  in  the  experimental 
design.  Several  types  of  supplemental  data  collection  techniques  can  be  used.  This  reference 
concentrates  on  the  graphical  rating  scales  that  are  most  often  used  to  collect  quantitative 
supplemental  data  in  human  factors  research. 

2.2.5  Topic  5.  Analysis  of  Nominal  Scale  Data 

Topic  5  deals  with  an  overview  of  major  supplemental  data  analysis  alternatives  that  can  be  used 
with  nominal  scale  data.  Only  a  sample  of  nonparametric  analyses  covering  the  most  common 
techniques  used  in  human  factors  and  ergonomics  research  is  presented  in  this  topic.  Siegel 
(Siegel  and  Castellan,  1988)  provides  a  detailed  discussion  of  all  of  these  techniques,  and  their 
formulae  and  notation  are  used  throughout  this  topic  for  easy  reference.  The  nonparametric 
techniques  in  this  reference  are  organized  around  between-subjects  and  within-subjects 
techniques  to  facilitate  an  easy  choice  of  nonparametric  analysis  procedures  for  nominal  data. 

2.2.6  Topic  6.  Analysis  of  Ordinal  Scale  Data 

Topic  6  deals  with  an  overview  of  four  nonparametric  analysis  alternatives  that  can  be  used  with 
ordinal  scale  supplemental  data.  The  four  procedures  described  in  this  reference  material  are 
often  used  in  human  factors  and  ergonomics  research.  Once  again,  Siegel  and  Castellan  (1988) 
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provide  a  detailed  discussion  of  each  of  these  techniques,  and  their  formulae  and  notation  are 
used  throughout  this  topic  for  easy  reference.  Similar  to  the  approach  followed  in  the  discussion 
on  nominal  data  analysis,  the  presentation  of  this  topic  is  organized  around  between-subjects  and 
within-subjects  techniques  to  facilitate  choice  of  nonparametric  analysis  procedures  for  ordinal 
data. 

2.2.7  Topic  7.  Summary  of  Supplemental  Data 

Topic  7  summarizes  the  major  points  discussed  in  section  2  of  the  reference  material  dealing 
with  supplemental  data.  First,  supplemental  data  collection  procedures  dealing  with  self  reports, 
questionnaires,  and  rating  scales  are  reviewed.  Second,  supplemental  data  analysis  techniques 
using  common  nominal  and  ordinal  scale  nonparametric  procedures  are  reviewed.  Third,  a 
three-step  process  for  dealing  with  supplemental  data  in  experimental  design  is  presented. 

2.2.8  Topic  8.  Introduction  to  ANOVA 

Topic  8  introduces  ANOVA  designs  by  discussing  their  advantages,  basic  terms,  and  the  three 
major  categories  of  ANOVA  designs  used  in  human  factors  research.  Next,  procedures  for 
specifying  the  underlying  statistical  model  that  describes  the  components  of  any  ANOVA  design 
are  presented.  The  introduction  ends  with  a  discussion  and  example  for  using  ANOVA  for 
statistical  hypothesis  testing  of  the  difference  between  two  treatment  means. 

2.2.9  Topic  9.  ANOVA  Summary  Table  Components 

Topic  9  provides  an  overview  of  the  computational  aspects  of  any  between-subjects,  within- 
subjects,  and  mixed-factors  ANOVA  design  used  in  human  factors  research.  The  subsections  are 
organized  around  the  five  major  components  of  the  ANOVA  Summary  Table  used  for  listing  the 
results  of  an  ANOVA.  Rather  than  derive  formulae  for  calculating  each  component, 
computational  procedures  and  algorithms  are  provided.  First,  each  component  of  the  summary 
table  is  discussed  separately.  Then  conventions  for  stating  the  complete  ANOVA  summary  table 
are  presented  for  each  of  the  three  major  categories  of  ANOVA  experimental  designs  used  in 
human  factors  research. 

2.2.10  Topic  10.  Between-Subjects  ANOVA  Designs 

Topic  10  covers  the  construction  and  computational  details  of  the  first  of  the  three  major 
categories  of  ANOVA  designs  used  in  human  factors  and  ergonomics  research.  Between- 
subjects  designs  are  discussed  in  terms  of  one-factor,  two-factor,  and  n- factor  designs.  The 
procedures  for  determining  the  sum  of  squares  computational  fonnulae  for  any  ANOVA  are 
presented  in  the  discussion  of  one-factor  designs.  The  concept  of  an  interaction  is  presented  in 
two-factor  designs.  Generalizations  for  constructing  and  analyzing  any  between-subjects 
ANOVA  design  are  summarized  under  n-factor  designs.  Computational  examples  are  provided 
for  both  a  one-way  and  a  two-way  between-subjects  design.  A  summary  listing  of  all  the 
ANOVA  procedural  rules  and  algorithms  for  conducting  any  ANOVA  analysis  in  human  factors 
research  is  provided  at  the  end  of  this  topic. 
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2.2.11  Topic  11.  Analysis  of  Comparisons  and  Interactions 

Topic  1 1  covers  analytical  techniques  that  can  be  used  to  isolate  the  form  or  nature  of  the  main 
effects  and  interactions  that  are  significant  in  the  overall  ANOVA.  Basically,  these  procedures 
deal  with  multiple  paired  comparisons  of  various  treatment  means.  First,  this  topic  covers 
comparisons  and  provides  examples  of  both  planned  and  unplanned  comparison  procedures. 
Second,  special  analytical  procedures  in  addition  to  paired  comparisons  are  covered,  and 
example  computations  are  provided  for  the  analysis  of  interactions.  Although  these  procedures 
are  demonstrated  using  between-subjects  designs,  these  same  techniques  are  appropriate  for 
within-subjects  and  mixed-factors  designs  covered  in  the  next  two  topics. 

2.2.12  Topic  12.  Within-Subjects  ANOVA  Designs 

Topic  12  covers  within-subjects  ANOVA  designs  in  which  each  subject  receives  every  treatment 
condition  in  the  human  factors  experiment.  Basic  configurations  of  one-  and  two-factor  designs 
as  well  as  generalizations  to  n-factor  repeated  measures  designs  are  covered.  The  advantages  of 
within-subjects  designs  are  summarized  as  compared  to  between-subjects  designs.  Additional 
considerations  in  using  repeated  measures  are  discussed,  including  the  homogeneity  of 
covariance  assumption,  balancing  techniques  for  controlling  possible  confounding  effects  of 
treatment  orders,  and  the  effect  of  differential  transfer. 

2.2.13  Topic  13.  Mixed-Factors  ANOVA  Designs 

Topic  13  covers  the  basic  configuration  and  analytical  procedures  used  in  mixed-factors 
ANOVA  designs,  which  comprise  the  third  major  category  of  ANOVA  designs.  Mixed-factors 
designs  are  composed  of  both  between-subjects  and  within-subjects  factors.  This  type  of 
ANOVA  design  is  often  referred  to  as  split-plot  designs  in  scientific  literature.  These  designs  are 
used  quite  frequently  in  human  factors  and  ergonomic  research  due  to  the  nature  of  the 
independent  variables  being  investigated  in  the  experiment.  Consider  a  training  research  study 
that  investigates  training  methods  and  practice  trials.  The  researcher  must  manipulate  the 
training  condition  variable  as  a  between-subjects  variable  because  subjects  cannot  return  to  a 
beginning  level  of  knowledge  when  provided  with  alternative  training.  On  the  other  hand, 
practice  trials  in  this  training  experiment  must  be  manipulated  as  a  within-subjects  factor  since 
each  subject  receives  multiple  trials.  Consequently,  a  mixed-factors  design  is  needed.  This  topic 
also  provides  a  list  of  considerations  that  the  research  should  address  when  using  mixed-factors 
designs. 

2.2.14  Topic  14.  Summary  of  Basic  ANOVA 

Topic  14  provides  a  brief  summary  review  and  roadmap  of  basic  ANOVA  in  terms  of 
fundamental  considerations,  ANOVA  design  classification,  ANOVA  generalizations,  and  steps 
in  the  overall  ANOVA  design  and  data  interpretation  process.  Details  on  these  issues  are 
provided  in  specific  topics  referenced. 
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2.2.15  Topic  15.  Introduction  to  Advanced  ANOVA 

Topic  15  introduces  topics  in  advanced  ANOVA  that  expand  upon  the  basic  ANOVA 
procedures,  rules,  and  algorithms  already  described  in  section  3  of  the  reference  material.  Two 
examples  of  extending  basic  ANOVA  (i.e.,  Quasi-F  ratios  and  Randomized  Blocks  Designs)  are 
presented  in  detail.  In  addition,  this  introduction  provides  an  overview  of  several  special  purpose 
ANOVA  design  and  analysis  procedures  that  satisfy  various  constraints  present  in  human  factors 
and  ergonomics  research.  Detailed  discussions  of  these  procedures  are  presented  in  subsequent 
topics. 

2.2.16  Topic  16.  Hierarchical  ANOVA  Designs 

Topic  16  covers  hierarchical  ANOVA  designs  that  include  nested  factors  of  interest.  This  topic 
describes  both  the  basic  layout  and  the  ANOVA  computations  involved  in  this  class  of 
experimental  designs.  These  basic  procedures  can  be  generalized  to  higher-order  hierarchical 
designs,  and  the  use  of  hierarchical  designs  in  human  factors  and  ergonomics  research  is 
discussed.  This  topic  ends  with  a  general  summary  of  the  considerations  of  using  hierarchical 
designs  in  human  factors  and  ergonomics  research. 

2.2.17  Topic  17.  Blocking  ANOVA  Designs 

Topic  17  deals  with  ANOVA  experimental  designs  that  can  be  used  to  control  nuisance  variables 
that  are  confounded  with  the  factors  of  interest  in  an  experiment.  These  confounding  factors 
may  include  the  effect  of  repeated  testing  sessions  or  experimenter  bias  resulting  from  using 
several  experimenters  in  data  collection.  Confounding  is  controlled  through  blocking  designs 
where  blocks  represent  the  nuisance  variables.  Through  the  use  of  modular  representation,  the 
exact  nature  of  the  block  confounding  with  specific  treatment  effects  of  interest  can  be 
determined  by  the  experimenter  in  order  to  choose  an  experimental  design  to  avoid  confounding 
blocks  with  effects  of  major  interest  in  a  factorial  ANOVA  design.  The  use  of  blocking  is 
demonstrated  through  2k  factorial  designs  and  extended  to  pseudo-factor  blocking. 

2.2.18  Topic  18.  Fractional  Factorial  ANOVA  Designs 

Topic  18  deals  with  fractional  replicates  of  full  factorial  ANOVA  designs.  When  large-scale 
factorial  designs  are  used  in  human  factors  and  ergonomics  research,  constraints  such  as  time, 
money,  and  equipment  availability  may  make  the  complete  factorial  design  unfeasible  and  data 
collection  must  be  restricted  to  a  portion  of  the  design.  Obviously,  some  effects  in  the  full 
factorial  design  cannot  be  evaluated  and  some  effects  will  be  confounded  with  others  if  data  are 
not  collected  on  the  entire  design.  The  experimenter  must  select  a  subset  of  the  complete  design 
that  yields  the  most  useful  data.  Fractional-factorial  designs  are  ANOVA  designs  in  which  only 
a  fractional  portion  of  the  complete  factorial  designs  are  observed.  Fractional  replications  of  2k, 
3k,  and  5k  designs  can  be  constructed  through  modular  representation.  Most  often  2k"p  fractional 
replicates  are  used  in  human  factors  research  to  avoid  subsequent  confounding  of  partial  effects 
that  are  present  in  3k  and  5k  designs.  Alternatively,  a  subset  of  fractional-factorial  designs,  called 
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Latin  Square  designs,  can  be  used  to  specify  the  subset  of  treatment  conditions  to  observe  when 
the  experimenter  is  only  interested  in  testing  the  main  effects  of  three  factors  of  interest. 
Consequently,  this  topic  describes  both  2k'p  fractional  replicates  and  Latin  Square  designs. 

2.2.19  Topic  19.  Analysis  of  Covariance  (ANCOVA) 

Topic  19  deals  with  an  analytical  technique  for  reducing  the  effect  of  a  covariate  to  increase  the 
sensitivity  of  the  F-test  on  effects  of  interest  to  the  experiment.  The  covariate  is  correlated  with 
the  dependent  variable  and  its  effect  is  removed  through  simple  linear  regression.  Both 
calculations  of  correlation  and  simple  regression  are  described  as  the  basic  components  of 
ANCOVA.  Basic  computations  in  ANCOVA  and  subsequent  interpretations  of  results  are 
described  in  this  topic. 

2.2.20  Topic  20.  Summary  of  Advanced  ANOVA 

Topic  20  summarizes  the  advanced  ANOVA  techniques  described  in  the  reference  material. 

This  topic  provides  a  summary  of  design  constraints  and  describes  a  process  for  addressing  them. 

2.2.21  Topic  21.  Introduction  to  Empirical  Models 

Topic  21  provides  an  overview  of  quantitative  models  with  an  emphasis  on  empirical  model 
building  developed  through  efficient  experimental  design.  Empirical  models  are  descriptive 
models  of  human  behavior  based  on  results  obtained  through  one  or  more  controlled  experiments 
that  can  be  used  to  predict  human  perfonnance  in  complex  systems.  These  empirical  models  can 
assist  the  human  factors  specialist  in  conducting  design  tradeoffs  of  critical  interface  parameters. 

2.2.22  Topic  22.  Multiple  Regression 

Topic  22  provides  an  overview  of  multiple  regression  procedures  used  to  generate  empirical 
models  that  involve  more  than  one  factor.  First,  multiple  linear  regression  is  described  to 
demonstrate  the  calculations  involved  when  considering  more  than  on  factor  in  the  empirical 
model.  Both  classical  and  modern  regression  procedures  for  selecting  the  best  multiple 
regression  equation  are  described.  Next,  polynomial  regression  is  discussed  as  the  general  form 
for  stating  empirical  models  in  human  factors  research. 

2.2.23  Topic  23.  Central-Composite  Designs  (CCDs) 

Topic  23  describes  experimental  designs  that  can  be  used  to  collect  data  for  building  second- 
order  empirical  models.  Specifically,  this  topic  focuses  on  CCDs  that  were  developed  to  explore 
response  surfaces  using  empirical  models.  The  background,  specification,  analysis,  and  an 
example  of  a  CCD  along  with  a  comparison  to  alternative  second-order  experimental  designs  are 
discussed  in  this  topic. 
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2.2.24  Topic  24.  Sequential  Experimentation 

Topic  24  incorporates  the  previous  topics  into  an  overall  strategy  for  conducting  research  on 
complex  systems  often  addressed  in  human  factors  and  ergonomics  research.  Due  to  nature  of 
complex  research,  the  experimenter  can  conduct  a  series  of  small  independent,  but  interrelated, 
experiments  rather  than  one  large  complex  study.  This  topic  describes  general  strategies  for 
conducting  complex  experiments  and  specifically  draws  upon  considerations  made  in  response 
surface  methodology  for  conducting  a  series  of  small,  interrelated  studies.  A  general  paradigm 
for  conducting  sequential  research  is  presented  along  with  a  detailed  example  of  using  this 
paradigm  in  human  factors  research  that  resulted  in  a  set  of  guidelines  for  sequential 
experimentation.  The  topic  concludes  with  a  description  of  combining  sequential  experiments 
into  a  common  database  that  incorporates  the  results  of  several  experiments.  This  database  can 
then  be  interrogated  to  generate  integrated  empirical  models  across  experiments. 

2.2.25  Topic  25.  Summary  of  Empirical  Models 

Finally,  topic  25  summarizes  the  empirical  model  building  techniques.  The  purpose  and 
characteristics  of  empirical  model  building  experiments  are  reviewed,  the  major  components  of 
empirical  models  are  listed,  and  a  sequential  experimentation  process  for  human  factors  research 
is  provided. 

2.3  CADRE  Bookmarks 

The  PDF  file  in  CADRE  is  heavily  bookmarked  around  the  nine  expandable  major  bookmark 
tabs  shown  in  table  2,  which  include  over  500  bookmarks  when  fully  expanded.  The  bookmarks 
were  designed  to  provide  the  CADRE  user  with  alternative  paths  for  rapid  location  of 
information  within  the  tool. 


Table  2.  Major  bookmark  tabs  in  CADRE. 


Bookmark  Tab  1.  Forward 

Bookmark  Tab  2.  List  of  Topics 

Bookmark  Tab  3.  Notes  on  Computational  Examples 

Bookmark  Tab  4.  Experimental  Design  Procedures 

Bookmark  Tab  5.  Statistical  Analysis  Procedures 

Bookmark  Tab  6.  Appendix  on  SAS  Examples 

Bookmark  Tab  7.  Acknowledgements 

Bookmark  Tab  8.  References/Supplemental  Readings 

Bookmark  Tab  9.  Subject  Index 


The  first  two  major  bookmark  tabs  shown  in  table  2  deal  with  selecting  specific  content  in 
CADRE.  Bookmark  1  is  a  single  bookmark  to  an  overview  for  using  the  CADRE  tool. 
Bookmark  tab  2  expands  to  provide  immediate  access  to  various  subsections  of  the  reference 
material  for  each  topic  in  the  table  of  contents  listed  in  table  1 . 
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Two  major  tab  sections  shown  in  table  2  expand  to  bookmarks  that  provide  direct  access  to 
various  experimental  design  and  statistical  analysis  procedures  described  in  the  reference 
material.  Experimental  design  procedures  in  bookmark  tab  4  deal  with  issues  such  as  research 
stages,  critical  research  methods,  basic  and  advanced  ANOVA  design  selection,  paradigms  and 
guidelines  for  sequential  experimentation,  supplemental  data  collection,  and  components  of 
research  reports.  Bookmark  tab  5  deals  with  various  statistical  analysis  procedures  including 
basic  statistics;  various  basic  and  advanced  ANOVA  computations;  simple,  multiple  linear,  and 
polynomial  regression  analysis;  and  empirical  model  building  procedures. 

Bookmark  tabs  3  and  6  expand  to  the  40  computational  examples  provided  throughout  CADRE 
(see  section  2.4).  Bookmark  tab  3  expands  to  a  discussion  of  statistical  computations  for  each 
example  in  the  reference  material.  Alternatively,  bookmark  tab  6  provides  immediate  access  to 
the  SAS  program  appendix  (Slater  and  Williges,  2006)  that  describes  all  the  SAS  (SAS,  2004) 
statistical  analysis  procedures  used  in  CADRE. 

The  final  three  major  bookmark  tabs  shown  in  table  2  deal  with  additional  reference  material. 
Bookmark  tab  7  is  a  single  bookmark  to  the  tool  acknowledgements.  Bookmark  tab  8  includes 
an  alphabetical  listing  of  all  references  used  in  the  CADRE  tool,  and  can  be  expanded  to  provide 
bookmarks  to  supplemental  reading  for  book  chapters  and  articles  in  the  scientific  literature  to 
elaborate  upon  the  reference  material  presented  for  each  topic  listed  in  table  1.  Finally, 
Bookmark  tab  9  expands  to  an  extensive  subject  index  arranged  alphabetically  for  easy  access  to 
various  topics  in  the  reference  material. 

2.4  CADRE  Links  and  Hyperlinks 

The  PDF  file  of  CADRE  also  supports  links  between  the  computational  descriptions  of  the 
examples  in  the  reference  material  with  the  40  examples  in  the  Slater  and  Williges  (2006) 
appendix,  as  shown  in  table  3.  The  reference  material  describes  the  various  steps  in  the 
statistical  calculation;  whereas,  the  use  of  the  SAS  (2004)  statistical  package  for  conducting  the 
analysis  is  described  in  the  appendix.  Each  example  in  the  appendix  provides  a  problem 
description,  a  SAS  input  file,  a  SAS  output  file,  and  output  explanation.  Notes  are  provided 
throughout  the  appendix  to  describe  tailored  modifications  when  using  various  SAS  procedures. 
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Table  3.  Hyperlinked  SAS  examples  in  CADRE. 


Example  1 .  Interval  Estimation 

Example  2.  Single-Sample  t-Test 

Example  3.  Between-Subjects  t-Test 

Example  4.  Within- Subjects  t-Test 

Example  5.  Chi-Square  Goodness  of  Fit  Test 

Example  6.  Chi-Square  Test  of  Independence  (2x2  Contingency  Table) 

Example  7.  Chi-Square  Test  of  Independence  (RxC  Contingency  Table) 

Example  8.  Chi-Square  Test  of  Independence  (Two  Additive  2x2  Partitions) 

Example  9.  McNemar  Change  Test 

Example  10.  Cochran  Q  Test 

Example  11.  Kolmogorov-Smirnov  Tests 

Example  12.  Kruskal- Wallis  One-Way  ANOVA 

Example  13.  Wilcoxon  Signed  Ranks  Test 

Example  14.  Friedman  Two-Way  ANOVA 

Example  15.  One-Factor,  Between-Subjects  ANOVA 

Example  16.  Two-Factor,  Between-Subjects  ANOVA 

Example  17.  Planned  Comparisons 

Example  18.  Unplanned  Comparisons 

Example  19.  Analysis  of  Interactions 

Example  20.  One-Factor,  Within- Subjects  ANOVA 

Example  21.  Two-Factor,  Within-Subjects  ANOVA 

Example  22.  Geisser-Greenhouse  and  Huynh-Feldt  Corrections 

Example  23.  Testing  Order  Effects  in  Balanced  Latin  Squares 

Example  24.  Within-Subjects  and  Between-Subjects  Design  Comparison 

Example  25.  Two-Way,  Mixed-Factors  ANOVA 

Example  26.  Complete  Hierarchical  Between-Subjects  Design 

Example  27.  Partial  Hierarchical  Between-Subjects  Design 

Example  28.  Simple  Blocking  of  2k  Within-Subjects  Design 

Example  29.  Complex  Blocking  of  2k  Within-Subjects  Design 

Example  30.  One-Half  Replicate  of  24  Between-Subjects  Design 

Example  3 1 .  4x4  Latin  Square  Designs 

Example  32.  Linear  Correlation  Coefficient 

Example  33.  Alternative  Linear  Correlations 

Example  34.  Simple  Linear  Regression 

Example  35.  Analysis  of  Covariance  (ANCOVA)  One-Way,  Between-Subjects  Design 

Example  36.  Multiple  Linear  Regression 

Example  37.  Best  Regression  Equation 

Example  38.  Polynomial  Regression 

Example  39.  Orthogonal,  Between-Subjects,  Central-Composite  Design 

Example  40.  Blocked,  Within-Subjects,  Central-Composite  Design 

Each  of  the  40  statistical  examples  in  the  Slater  and  Williges  (2006)  appendix  is  also  hyperlinked 
to  version  9.1.3  of  the  SAS  (2004)  statistical  analysis  package.  If  the  SAS  interactive  statistical 
package  is  active  on  a  user’s  computer,  the  CADRE  tool  can  be  directly  linked  to  the  SAS 
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program  file  for  any  example  covered  in  the  appendix.  These  programs  also  can  be  used  as  a 
guide  for  analyzing  an  experimenter’s  specific  data  set  by  simply  modifying  the  names  and 
numbers  of  independent  variables  and  the  specific  data  set  of  the  dependent  variables. 


3.  The  Focus  of  CADRE 


The  major  focus  of  CADRE  is  on  understanding  the  various  techniques  and  processes  considered 
in  applied  experimental  design  by  researchers  who  have  a  basic  knowledge  in  statistics  and 
experimental  design.  In  order  to  conduct  complex  system  research  experiments,  researchers 
must  consider  a  variety  of  experimental  design  and  analysis  techniques.  The  CADRE  tool 
features  a  review  of  the  design  construction,  assumptions,  and  basic  computations  in  a  variety  of 
experimental  design  techniques  with  an  emphasis  on  ANOVA  procedures  and  related  designs. 
Rather  than  mathematical  derivations,  algorithms  and  guidelines  are  used  to  promote 
understanding.  Saying  that,  references  to  the  scientific  literature  are  provided  throughout 
CADRE  for  details  on  the  various  techniques  and  their  use  in  complex  applied  research 
problems.  Most  researchers  use  statistical  packages  for  analysis,  so  the  focus  is  on  the  use  and 
interpretation  of  SAS  rather  than  hand  calculations.  Examples  of  hand  calculations  are  provided 
in  CADRE  to  promote  a  better  understanding  of  the  various  experimental  design  techniques. 

The  process  of  choosing  various  techniques  in  applied  experimental  design  is  also  a  focus  of 
CADRE.  This  includes  considering  the  various  stages  in  the  research  process;  the  supplemental 
data  collection  and  analysis  process;  the  process  of  empirical  model  building;  and  a  process  for 
conducting  sequential  experimentation. 

CADRE  represents  one  example  of  a  computer-based  desktop  interactive  tool  that  can  assist 
researchers  in  conducting  complex  systems  research.  This  tool  can  be  expanded  to  incorporate 
additional  experimental  design  procedures  and  additional  statistical  analyses  packages  that  can 
hyperlinked  to  the  reference  material  in  CADRE.  This  tool  needs  to  be  widely  available  to  the 
research  community  so  that  improvements  based  on  usage  can  be  made  to  both  the  content  and 
interactive  nature  of  CADRE. 

In  addition,  the  approach  used  in  developing  the  CADRE  tool  can  be  extended  to  other  desktop 
tools  to  assist  the  human  factors  researcher.  CADRE  can  be  considered  the  first  prototype  in  a 
series  of  tools.  Other  computer-based  tools  addressing  a  variety  of  research  methods  (e.g., 
multivariate  analysis,  survey  procedures,  and  cognitive  task  analysis),  system  design  methods 
(e.g.,  focus  groups,  design  guidelines,  specifications,  and  requirements),  and  evaluation  methods 
(e.g.,  usability  assessment,  operator  workload  evaluation,  situation  awareness,  and  crew  resource 
management)  can  be  added  to  this  family  of  tools  to  be  used  by  human  factors  professionals. 
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4.  Conclusions 


Research  on  Army  C2  systems  requires  knowledge  and  use  of  complex  experimental  design  and 
analysis.  The  CADRE  tool  can  assist  in  designing  effective  experiments  that  allow  data 
collection  requiring  minimum  effort  and  maximum  effectiveness  of  results.  Sections  4  and  5  of 
CADRE  deal  specifically  with  fractional-factorial  designs,  polynomial  regression,  empirical 
model  building,  and  sequential  experimentation,  which  are  particularly  appropriate  for 
investigating  emerging  C2  systems. 

The  studies  that  provided  impetus  to  spawn  the  development  of  the  CADRE  tool  were  part  of 
ongoing  efforts  to  develop  better  understandings  of  how  to  structure  socio-technical  work 
environments.  These  conditions  are  characterized  where  teams  function  in  a  high  stress,  time 
dependent,  decision-oriented  work  setting.  In  these  situations,  the  goal  is  to  optimize  human 
cognitive  performance  in  this  type  of  work  domain,  such  as  can  be  found  in  military  C2  systems. 
This  work  is  especially  relevant  for  studies  of  systems  not  yet  in  existence,  such  as  the  Future 
Combat  System  (FCS)  and  Objective  Force  Warrior  (OFW). 
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Acronyms  and  Abbreviations 


ADA 

Advanced  Decision  Architectures 

ANCOVA 

Analysis  of  Covariance 

ANOVA 

analysis  of  variance 

ARL 

U.S.  Army  Research  Laboratory 

C2 

command  and  control 

CADRE 

Computer-Aided  Design  Reference  for  Experiments 

CCD 

Central-Composite  Designs 

FCS 

Future  Combat  System 

HRED 

Human  Research  and  Engineering  Directorate 

OFW 

Objective  Force  Warrior 

PDF 

Postscript  Definition  File 

Virginia  Tech 

Virginia  Polytechnic  Institute  and  State  University 

16 


No.  of 

Copies  Organization 


No.  of 

Copies  Organization 


1  ADMNSTR 

ELECT  DEFNS  TECHL  INFO  CTR 

ATTN  DTIC  OCP  (ELECTRONIC  COPY) 
8725  JOHN  J  KINGMAN  RD  STE  0944 
FT  BELVOIR  VA  22060-6218 

1 0  ARMY  RSRCH  LABORATORY-HRED 

ATTN  AMSRD  ARL  HR  MV 
HQUSAOTC  S  MIDDLEBROOKS 
91012  STATION  AVE  RM  348 
FT  HOOD  TX  76544-5073 

1  US  ARMY  RSRCH  LAB 

ATTN  AMSRD  ARL  Cl  OK  TP 
TECHL  LIB  T  LANDFRIED 
BLDG  4600 
APG  MD  21005-5066 

1  US  ARMY  RSRCH  LAB 

ATTN  AMSRD  ARL  Cl  OK  PE 
TECHL  PUB 

ATTN  AMSRD  ARL  Cl  OK  TL 
TECHL  LIB 

ATTN  IMNE  ALC  IMS  MAIL  & 
RECORDS  MGMT 
ADELPHI  MD  20783-1197 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  M 
DR  M  STRUB 

6359  WALKER  LANE  SUITE  100 
ALEXANDRIA  VA  22310 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  ML  J  MARTIN 
MYER  CENTER  RM  2D3 1 1 
FT  MONMOUTH  NJ  07703-5601 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MZ 
A  DAVISON 

199  E  4TH  ST  STE  C  TECH  PARK  BLG  2 
FT  LEONARD  WOOD  MO  65473-1949 


1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MJK 
J  HANSBERGER 

JFCOM  JOINT  EXPERIMENTATION  J9 
JOINT  FUTURES  LAB 
115  LAKE  VIEW  PARKWAY  SUITE  B 
SUFFOLK  VA  23435 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MQ  M 
R  FLETCHER 

US  ARMY  SBCCOM  NATICK 
SOLDIER  CTR 

AMSRD  NSC  SS  E  BLDG  3  RM  341 
NATICK  MA  01760-5020 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MY 
DR  J  CHEN 

12423  RESEARCH  PARKWAY 
ORLANDO  FL  32826 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MS 
C  MANASCO 

SIGNAL  TOWERS  118  MORAN  HALL 
FORT  GORDON  GA  30905-5233 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MU 
M  SINGAPORE 

6501  E  1 1  MILE  RD  MAIL  STOP  284 
BLDG  200A  2ND  FL  RM  2104 
WARREN  MI  48397-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MF 
C  HERNANDEZ 
BLDG  3040  RM  220 
FORT  SILL  OK  73503-5600 


1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MD  T  COOK 
BLDG  5400  RM  C242 
REDSTONE  ARSENAL  AL  35898-7290 
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No.  of 

Copies  Organization 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MW  E  REDDEN 
BLDG  4  ROOM  332 
FT  BENNING  GA  31905-5400 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MN 
R  SPENCER 
DCSFDI  HF 

HQ  USASOC  BLDG  E2929 
FORT  BRAGG  NC  28310-5000 

1  ARMY  G1 

ATTN  DAPE  MR  B  KNAPP 

300  ARMY  PENTAGON  ROOM  2C489 

WASHINGTON  DC  20310-0300 

1  DIRECTOR 

US  ARMY  RSCH  LABORATORY 
ATTN  AMSRD  ARL  Cl  OK  TP 
S  FOPPIANO 
BLDG  459 

APGMD  21005-5066 

1  US  ARMY  OPERATIONAL  TEST  COM 

METHODOLOGY  AND  ANALYSIS  DIR 
ATTN  TEOT  MA 
WAYLAND  SMITH 
91012  STATION  AVENUE 
FORT  HOOD  TX  76544-5068 

1  US  ARMY  OPERATIONAL  TEST  COM 

TRANSFORMATION  TECH  DIR 
ATTN  TEOT-TT 
GAYLE  SHULL 
91012  STATION  AVENUE 
FORT  HOOD  TX  76544-5068 


TOTAL  38(1  Elec,  37  HCs) 


